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FOREWORD 



The Manpower & Educational Systems Technical Area of the Army 
Research Institute for the Behavioral and Social Sciences (ARI) perfon 
research and development in areas that include educational technology 
and training simulation with applicability to military training. Of 
special interest is research in the area of computer-based systems for 
maintenance training. The development and implementation of such 
systems is seen as a means of reducing ..training time and costs by 
providing more highly individualized training than would be otherwise 
possible, while at the same time reducing the need for operational 
equipment for training. 

The report provides an annotated bibliography of research direct; 
related to human problem-solving behavior in fault diagnosis (troubled- 
shooting) tasks. It provides an information source to assist planners 
of future research efforts in this area. 

This research is responsive to the requirements of RDT&E Project 
2Q161102B74F, "Basic Research in the Behavioral and Social Sciences." 
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AN ANNOTATED SELECTIVE BIBLIOGRAPHY ON HUMAN PERFORMANCE IN FAULT 
DIAGNOSIS TASKS 



BRIEF 



Requirement : 



T.n conjunction with an ongoing program of research on the use of 
comp«s:er simulations for fault diagnosis trainiSg, a thorough review of 
the existing literature was necessary to provide historical input and to 
Identify current trends in fault diagnosis work. The results of this 
search are presented here as a selective annotated bibliography. 

Procedure: 

The search procedure consisted of on-line computer search of the 
^^r,i2o^®"° °^ Psychological Abstracts and ERIC Docutaents through 
mid-1978. Manual searches used references from reports as well as from 
the card catalog of the University of Illinois library. In addition 
experts m the area of fault diagnosis were sent the references resulting 
from the computer and manual searches and asked to suggest any additional 
references that should be included. 

Findings: 

The references revealed by the search procedure were predominately 
...from the areas of engineering, psychology, and education. Three major 
categories of materials are presented. These consist of (1) research 
reports, (2) manuals for fault diagnosis, and (3) material indirectly 
related to fault diagnosis, such as medical diagnosis and mathematical 
problem solving, or items directly related to fault diagnosis that are 
not research reports. 

Utilization of Findings: 

This annotated bibliography will serve as' an information source 
for planners of future R&D efforts in this area. 
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INTRODUCTION 

The authors and several colleagues at the University of 
Illinois are investigating the use of computer simulations for 
fault diagnosis training. This work runs the gamut from 
context-free simulations and context-specific simulations into 
the live environment of real equipment troubleshooting. This 
research, supported by the U.S. Army Research Institute for the 
Behavioral and Social Sciences, includes the development of 
models of human fault diagnosis performance and ultimately an 
intelligent interactive training simulator. To this end, we have 
already run six experiments involving almost 160 subjects who 
solved approximately 13,000 fault diagnosis problems. 

As part of our work, we have prepared this annotated 

selected bibliography on human performance in fault diagnosis 
tasks. The sources in this bibliography are predominantly from 

the disciplines of engineering, psychology, and education. 

Although the primary emphasis is on the research literature, we 

have included a representative number of actual troubleshooting 
manuals from the automotive and aviation fields. 

. The literature search included several, phases, involving 
both on-line and manual searches. Computer searches of the past 
ten years of Psychological Abstracts and ERIC documents t-hrough 
mid-1 978 were performed. Manual searches were conducu^'d using 
bibliographies from various reports. Using the card catalog at 
the University of Illinois library, we obtained materials~ dating 
from, the early fifties. 5 
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The third phase of this literature search involved the 
solicitation of additional references from noted fault diagnosis 
and/or computer simulation experts. Over twenty such individuals 
were sent the list of references resulting from the first two 
phases of our literature search. "They were asked to skim the 
list of references and suggest any additional references that 
they felt to be essential for a selected bibliography on human 
performance in fault diagnosis tasks. Approximately one-third of 
those queried responded. We greatly appreciate their 
contributions. 

The materials in this bibliography are divided into the 
following three major areas: 

1 . Research Reports 

A. Primary Research 

B. Secondary Research 

2. Manuals for Fault Diagnosis 

3. Materials Indirectly Related to Fault Diagnosis 

Primary research reports include empirical and/or theoretical 
studies aimed directly at human problem solving behavior in fault 
diagnosis tasks. Secondary research reports refer to literature 
reviews. As- noted earlier, a representative set of training and 
field manuals is included in the "manuals" category. Indirectly 
related materials refer to studies of human behavior in related 
domains (e.g. medical diagnosis or problem solving in 
■': ^ , reports that directly relate to fault diagnosis. 

EKLC 
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but do not provide research results of a type necessary for the 
report to be' categorized as a "research report." Since the number 
of items that could be placed in the "indirectly related" 
ca.tegory was potentially enormous, we limited consideration to 
items -^jrha-t related to human behavior and particularly matched our 
interests. Thus, for example, the large body of literature on 
maintainability and automated fault diagnosis has been omitted 
from this bibliography. 

Where possible, we used the abstract that accompanied an 
item. However, we usually had to supplement the abstract 
considerably in order to make it self-explanatory. The indexing 
of each item was based -cm— a consensus of the authors.'' ^ 

We hope to periodically update this bibliography and 

re-issue it. To that end, we would appreciate suggestions and 

contributions. If possible, when submitting a suggestion for 

inclusion of a new item, please include a copy of the item. 



PRIMARY RESEARCH 



Bond, N.A., Jr.; Rigney, J.W, (1966) "Bayesian Aspects of 
Troubleshooting Behavior." Human Factor s. 8:377-383- 

Thirty-nine Navy technician trainees filled out a 
symptom-malfunction m'atrix on a blocking oscillator 
circuit. The particular sequence of checks used by each 
man on each problem wa s combined with his 
symptom-malfunction matrix, via a Bayesian algorithm, to 
yield computer estimates of failure likelihoods for each 
component. The computer program predicted actual 
parts-replacement behaviors in about half oT the cases. 
:v--^ -Those technicians who start out with valid 
symptom-malfunction matrices are more likely to resemble 
the Bayesian processor. Subject's troubleshooting 
behavior that was characterized as Bayesian correlated 
with higher class standing and" logger time between tests. 
It is suggested that symptom-malfunction matrix accuracy 
could be a training criterion. Students could be trained 
until they could produce a symptom-malfunction matrix to 
a high standard of physical validity. 

7 pages, 3 references 



Brooke, J. B.; Duncan, K.D.; Marshall, E.G. (1978) "Interactive 
Instruction in Solving Fault Finding Problems." International 
Journal Man-Machine Studies . .10:603-611. 

Thirt^y subjects were divided into two groups which solved 
60 trials, of fault diagnosis problems. One , group was 
given process information provided by the interactive 
software system when an incorrect diagnosis was made. At 
that point during the interaction, feedback regarding the 
possible failures was given ^to the subjects. Also, if 
subjects continued to ask for add itional information when 
the solution was obvious, the system informed subjects of 
tha fact that the failure could be identified. The 
control group had no feedback and were not informed about 
the correctness of their diagnosis. Analysis of variance 
showed a significant effect of training on learning. 

9 pages, 2 references 
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Brown, J,S.; Burton, R.R,; Bell, A.G, (1975) "SOPHIE: A Step 
Toward Creating a Reactive Learning Environment." International 
Journal of Man-Machine Studies . 7:675-696. 

This paper describes a fully operational AI-CAI system 
which incorporates artificial, intelligence techniques to 
perform question answering, hypothesis verification and 
theory formation activities in the domain of electronic 
troubleshooting. Much of its logical or inferencing 
capabilities are derived from uses of simulation models 
in conjunction with numerous procedural specialists. The 
system also includes a highly tuned structural parser for 
allowing the student to communicate in natural language. 
Students can test components in an electronic circuit, 
develop hypotheses regarding the fault, and obtain 
feedback concerning why a hypothesis is incorrect . 
Replacement of parts is allowed only when the student can 
answer correctly queries about why he thinks the parts 
should be replaced. Although the system is extremely 
large it is sufficiently fast to be thoroughly exercised 
in a training or classroon: environment. 



22 pages, 11 refej^ences 



Brown, J;S. ; Rubinstein, ; Burton, R, (1976) "Reactive 
Learning Environment for Computer Assisted Electronics 
Instruction." Lowry Air Force Base, CO: AFHRL-TR-76-68 . 

The development of several new computer based strategies 
for teaching .troubleshooting principles to electronics 
technicians is described. The report documents an 
experiment in which those materials were presented to 
student technicians to determine their attitudes toward 
the techniques, and to determine whether the resulting 
training resulted in improvement of their technical 
skills . Results in both cases were positive with 
students responding very favorably to the materials and 
with their performance improving quantitatively and 
qualitatively after the instruction. 

Seventeen college level students in an electronics 
curriculum were subjects in an experiment for evaluating 
alternative training and teaching methodologies on 
troubleshooting behavior . Subjects were divided into 
four groups, each group receiving a different sequence of 
teaching and training activities. Most of the evaluation 
is qualitative and based on extensive interviews and 
questionnaires. However, the fourth group was 

administered a pre-test for measuring basic understanding 
of power supply troubleshooting. This pre-test also 
included two faults to troubleshoot before the laboratory 
games began. A post-test was administered to measure the 
effect of computer-based activities on learning. 
Students' scores reflected their ability to make -correct 
fault hypotheses given a set of symptoms. Improvement in 
all subjects' scores was found to be significant at the 
.05 level. Also, subjects experienced an average 
decrease in task time by 41%, significant at the .05 
level. 

140 pages 
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Crawford, A.M.; Crawford, K.S. (1978) "Simulation of 
Operational Equipment with a Computer-Based System: A Low Cost 
framing Technology." Human Factors . 20 (2 ): 21 5-224 . 

•The feasibility and effectiveness of teaching performance 
Skills using a computer-based training (CBT) methodology 
were investigated. Graphic simulations of- the appearance 
and functions of a system in an anti-submarine aircraft 
were presented to students within an instructional 
framework. The objective was to- determine whether CBT 
could be used for low cost, part-task training. The 
subjects were new co-pilot trainees for the S-3A 
anti-submarine aircraft. The performance of the- 22 CBT 
students, as measured on a high fidelity simulator, was 
compared to that of 22 students who had gone through 
conventional training consisting of workbook study and 
hands-on practice in' the simulator. Results showed that 
CBT students performed the necessary skills as well 
before practice in the simulator as conventionally 
trained students could after this practice. Performance 
measures were the number of tasks completed and the time 
to completion. The authors felt that CBT was • an 
effective and low-cost means of part-task training of 
certain performance skills. 

10 pages, 11 references 
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Crooks, W.H,; Kuppin , M.A.; Freedy, A. (1977) "Application of 
Adaptive Decision Aiding Systems to Computer-Assisted 
Instruction: Adaptive Computerized Training System (ACTS)." 
Arlington, VA: U.S. Army Research Institute for the Behavioral 
and Social Sciences, PATR-1 028-77-1 . 

This report describes the Adaptive Computerized Training 
System (ACTS) which combines the techniques of circuit 
simulation , artificial intelligence , decision modeling, 
and adaptive computer- assisted instruction to provide 
training in decision making. The ACTS incorporates an 
adaptive computer program which learns a student's 
diagnostic value structure and uses this structure to 
train the student in practical decision making. This 
report describes the development and operation of the 
ACTS as it is applied to training electronics 
troubleshooting . 

Six college students with electronics backgrounds served 
as subjects, in solving 45 fault diagnosis problems. 
Subjects were divided into three groups receiving the 
following initial aiding: 1) the expert's three best 
alternatives were displayed to the subject ond after the 
subject's choice was made, additional feedback was 
provided by identifying the expert's best alternative; 
2) subjects first indicated alternatives before expert's 
alternatives were displayed; and 3) no feedback or 
indication of expert^ s choice. Results indicate that 
those who received aiding or feedback solved problems at 
lower costs than subjects who received neither aiding nor 
feedback. Practice was shown to have a positive effect 
for all groups in terms of reducing costs. 

54 pages " :P" 
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Elliott, T.K. (1966) "A Comparison of Three Methods for 
Presenting Procedural Troubleshooting Information." Aerospace 

n^?i°^ D^^"''^!'^M^ Laboratories, Wright-Patterson Air Force Base, 
Ohio. Report No. AMRL-TR-66-1 91 . ocis,« , 

The effects of subject aptitude and performance aide mode 
Of presentation on the performance of procedural, 
between-stage troubleshooting tasks on a real piece of 
electronic equipment were studied. The study used 
nondecison aids presented in three modes, namely, an 
automatic retrieval of visual information, an automatic 
retrieval of audio information, and a manual retrieval of 
visual information. Two aptitude groups (Air Force 
electronic index UO-60 and 75-90) with no previous 
electronic training or experience were given from 3 to 5 
nours of task training before the experiment. Each 
subject solved 26 actual, and 11 synthetic problems. The ' 
study indicated no difference in effectiveness of aids 
using visual representations. Both were superior to 
audio presentations. The two aptitude groups were 
equally effective in performing troubleshooting tasks 
using nondecision aids. In comparison with a previous 
study using decision aids, the study showed nondecision 
aids produced superior performance on the same 
between-stage troubleshooting problems using similar 
subjects. 

38 pages, 10 references 
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Elliott, T.K,; Joyce, R,P, (1971) "An Experimental Evaluation 
of a Method for Simplifying Electronic Maintenance," Human 
Factors , 1 3 ( 3 ) : 21 7-227 . 

Two groups of subjects solved the same set of 13, 
troubleshooting and . repair problems in seven 
solid-state-circuit modules which contained as many as 
five stages each. Both groups used the same hand tools 
and test equipment. One group was composed of 41 
conventionally trained Air Force 5- and 7-level 
technicians who normally maintain such equipment as part 
of their jobs. The technicians used the same 
troubleshooting techniques they ordinarily used on their 
jobs, and they were provided with a performance aid 
resembling an Air Force technical order. The other group 
was composed of 20 high-school students with no prior 
training or experience in electronics. Their training 
for this study consisted of a 12-hour course in the use 
of hand tools and test equipment and in the us^e of the 
procedur alized troubleshooting aid evaluated in this 
study. The aid indicated which check to make based upon 
the outcome of previous checks. Using the proceduralized 
troubleshooting aid , the high-school students took 
significantly less troubleshooting time than did the 
experienced technicians using normal techniques; 
however, the technicians required significantly less 
repair time and made significantly fewer errors than did 
the students. These differences may be accounted for in 
part--by'~~di"f f ere'nce'^s^'"Tn"'^th"e^^ 

the two groups, and there were no differences between the 
two groups on a number of other measures. However, the 
difference in training time and, therefore, cost of 
training between the two groups was so great as to 
suggest the possibility that job-relevant training and 
proceduralization of the task can introduce substantial 
savings, even after the cost of developing the special 
performance aids - required by proceduralized 

troubleshooting is subtracted, 

11 pages, 10 references 
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Ely,. J.H-; Hall^ N.B-; Van Albert, C.E. (I960) "Coding 
Electronic' Equipment to Facilitate Maintenance," IRE Transactions 
on Human Factors in Electronics , HFE-1 (2) : 66-69- ' 

The aim of this study was to improve maintenance of. 
electronic equipment by determining what information to 
place on the equipment and developing techniques for its 
display. Trips to various communications installations 
were made in order to examine equipment, observe 
maintenance activities, examine manuals, and interview 
personnel. Detailed data were collected showing the 
marked variation between observed test point readings and 
those called for in maintenance manuals. Recommendations 
were developed concerning information to be displayed on 
prime electronic equipment, covering 1) designation of 
functional groupings, 2) identification of signal paths, 
3) identification and sequence for using test points, and 
^) presentation of historical information for each test 
point . These recommendations were evaluated by 
incorporating them on an oscilloscope and evaluating the 
performance of 18 technicians. Comparing this "coded" 
scope with an identical uncoded one showed that 
troubleshooting time was reduced on the average to 
approximately one-half that for the uncoded equipment. 

U pages, 3 references 



Finch, C.R. (1971) "Troubleshooting Instruction in 

Vocational-Technical Education via Dynamic Simulation •" 
Pennsylvania Department of Education,. Bureau of Vocational, 
Technical and Continuing Education. ERIC Document Reproduction 
Service ED 055 204. 

This study used a commercial real equipment simulator to 
train autbmechanics for fault diagnosis. The research 
also explored the attitude of the students and the 
instructors towards the simulator . The experimental 
group received simulator practice , while the control 
group received no treatment. Those students who 
practiced on the simulator had better troubleshooting 
performance on four of the six criteria measures. The 
instructors were satisfied with the 'simulator, and the 
students believed that it was at least as good as 
traditional classroom or laboratory instruction. 



107 pages, 21 references. 
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Finch, C.R. (1972) ''The Effectiveness of Selected 
Self-Instructional Approac'^.cs in Teaching Diagnostic Problem 
Solving; " The Journal of Educational Research ^ 65:219-223. 

This study examined the effects of three 
self -instructional approaches of teaching diagnostic 
problem-solving (troubleshooting) / Two of the approaches 
attempted to simulate student-equipment interaction via 
programmed and text instruction while a third approach 
utilized actual equipment. After forty-five community 
college students received an orientation to equipment 
~ operation, fifteen were randomly assigned to each of the 
three treatments. Criterion variables includ.ed 
troubleshooting knowledge , troubleshooting performance , 
and attitude toward instruction received. Analysis of 
covariance indicated that the equipment approach was 
superior to non-equipment simulation approaches in terms 
of troubleshooting^ performance scores. Correlational 
analyses of criterion variables provided some evidence 
that relationships between attitude and behavior were 
dependent upon the instructional approach used. 

5 pages, 10 references 



Freedy, A.; Crooks, Wv H. (1975) Use of an Adaptive Decision 

Model— — in Com pu t e r- A"s s rs te'd~ " — Tr a i" n i'n g" ^ ojn " Electronics 

Troubleshooting " Woodland Hills, CA: Perceptions IncTl April 

The report discusses a sylstem for C*omputerized Decision 
Training (CDT)'. This system has the capability to 
compare subjects' troubleshooting decisions with the 
optimal, as defined by "experts." The goal was to train 
for quality decision making rather than cookbook 
troubleshooting. The simulations are computer generated 
problems applicable to electronic circuits. The problems 
are presented on a CRT and permit the subject to: check 
symptoms, measure a component value, replace a module, or 
declare circuit to be fault free. The adaptive decision 
model was found to converge on the decision behavior of 
the students who quickly settle into consistent decision 
strategies. The authors' conclusion notes that there is 
a need for considerably more research in this area, 

9 pages, 17 references 
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Glaser, R. ; Damrin, D.E.; Gardner, F.M. (195U) "The Tab Item: 
A Technique for the Measurement of Proficiency in Diagnostic 
Problem Solving Tasks." Educational and Psychological 
Measurement . 14: 283-293. " — 

This work describes a method of fault diagnosis training 
relying on the "tab item." The tab item is a foldout 
cardboard flow chart which allows the troubleshooter to 
gather information by pulling tabs. This system provides 
the following: A symptom, check procedures, and a list 
of possible failures. This system therefore attmepts to 
reproduce the troubleshooting situation on paper. 
Measures for scoring troubleshooting performance which 
are discussed are: number of tests performed, number of 
redundant, tests, test relevancy, and test adequacy. 
Although this article offers no empirical evidence of the 
instructional value of the tab item, it does speculate as 
to its potential value across a broad spectrum of 
technical training. Additional uses of the Tab Test can 
be found in Standlee ( 1956) and Fattii ( 1956). 

10 pages 
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■Glass, A. A. (1967) "Problem-Solving Techniques and 

Troubleshooting Simulators in Training Electronic Repairmen." 
Doctoral Thesis. Columbia University. 

The purpose of this project was to study the practical 
and operational application of problem-solving concepts 
by introducing into . the regular training of military 
electronic repairmen two experimental methods of teaching 
troubleshooting and to evaluate their effectiveness in 
promoting success. These two experimental methods were: 
1. a series of recorded lectures dealing with problem 
solving methods, and 2. a series of paper and pencil 
troubleshooting simulators to provide the opportunity to 
apply problem solving techniques. There were two 
simulators, one commercial and the other custom-built. 
The main criteria for evaluation of the experimental 
variables were performance tests on actual electronic 
equipment (N=240). The method for administering and 
scoring the performance tests was the result of extensive 
pilot studies. The essence of the scoring method was to 
score for sequence of logic in troubleshooting, time, and 
solution success. The conclusions were: 1. Subjects 
receiving problem-solving lectures were superior on 
performance test to subjects not receiving them. 2^ 
Subjects using simulators were superior on performance 
criteria tests to those not using the devices, but only 
when the . custom-built device was used. 3. Subjects 
receiving a combination of lectures and devices were 

superror-^ on"^ per foTmanc'e""GrTte"ri^^^^ to" sub^jec'ts "not 

receiving the combination. 4. The use of the 
custom-built simulator produced superior performance 
criteria test scores as compared to subjects using the 
commercial version, only when the lectures were used as 
well . 

136 pages, 54 references 
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Goldbeck, R.A.; Bernstein, B.B.; Hillix, W.A.; Marx, M.A, 
(1957) "Application of the Half-Split Technique to 
Problem-Solving Tasks Journal of Experimental Psychology. 
53(5) : 330-338. 

An- experiment was designed (1) to test the hypothesis 
that the half-split instruction increased efficiency for 
simple problems and (2) to determine half-split 
efficiency for more . complex problems. A second 
experiment was designed to determine whether additional 
practice, -different methods of instruction, and more 
complex training materials would produce an improvement 
in the subject»s ability to arrive at a set of 
alternatives, and, if so , whether the half-split method 
would then be more efficient. 

In experiment I, 90 male university students participated 
as subjects and were instructed to replace a faulty unit 
in a troubleshooting task. .Thirty were assigned problems 
to each of three complexity levels. Based on scores 
obtained from a problem solving test, subjects were 
randomly assigned to a control group and a group which 
received explicit instructions regarding the application 
of the half-split technique. Solution time for the 
half-split group was significantly longer than he 
control group. Increasing complexity re-^ :\e.^ 
increase in the number of tests only for th-^:- subjects 
whp_ _sc,pred low_o.n_the problem-.sol-v4ng> .test — Resul-t-s~ 



suggest that the half-split technique is superior under 
certain conditions , such as with diagrams of low 
complexity where subjects apparently had little 
difficulty in finding the trouble-possible units, and 
where subjects avoided incorrect replacements. 

Experiment II was an attempt to improve the effectiveness 
of the half-split method by improving instruction and 
giving extended practice. Only problems of the highest 
level of complexity were used. Forty male university 
students participated and were randomly assigned to four 
training groups. The deductive half-split group was like 
the previous half-split group but given more practice. 
The instructions and practice given the deductive 
half-split group were more effective in reducing the 
number of tests than were the less extensive instructions 
in Experiment I. Results of both experiments indicate 
that the empirical superiority of the half-split 
technique is a function of the nature of the task, and 
the degree and type of training given the- t roubleshooter . 
From these experiments, it is now clear that future 
research on training methods should emphasize the more 
basic deductive phase rather than the relatively 
mechanical half-splitting aspect of the process. 

9 pages, 3 references 
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Landa, S, (1972) CATTS: Computer-Aided Training in 
Troubleshooting . (R018-PR) Santa Monica , CA: Rand Corporation • 
ERIC Document Reproduction Service No. ED 066 888, 

The Rand Corporation's Programmer-Oriented Graphics. 
Operation (POGO) was used in the design, implementation 
and testing of a computer- assisted instruction course to 
train airmen in malfunction diagnosis - CATTS (Computer 
Aided Training in Troubleshooting). The , design of the 
course attempted to reduce the problems of computer 
graphics for both instructor and student. The 
observations of Air Force instructors using the system 
suggest that the system is workable. No empirical 
evidence was gathered. The report emphasizes the POGO 
software more than troubleshooting. 

108 pages, 11 references 



Mallory, W.J.; Elliott, T.K. (December 1978) "Measuring 
Troubleshooting Skills Using Hardware-Free Simulation." Air Force 
Human Resources Laboratory, AFHRL-TR-78-^i7 . 

The hypothesis formulated was that troubleshooting 
performance on simulation would be predictive of the same 

.t.a s k s_o.n _r.e a 1 -equipment S i-x-t e e n— i n s t r u ct o r-s—a n d- f-i-f-t e e n 

beginning students in a precision measurement equipment 
course we re given the same problems wi th two treatment 
levels. Problems were solved for criterion of job sample 
test (JST) and simulation or symbolic performance test 
(SPT). The task in each case was to locate the fault at 
the stage level rather than component level. Performance 
measures included mean percent correct solution, time to 
complete problems, and number of steps to completion. 
Statistically significant correlations indicate a high 
degree of similarity between the job sample test and 
symbolic performance test. An algorithm is presented 
which i3 used to evaluate problem difficulty. 
Correlation ; of the hypothesis problem difficulty with 
empirical problem difficulty (obtained by using test 
results of subjects) did not produce significant 
agreement. It was noted that the problem difficulty 
algorithm was insensitive to feedback loops which 
occurred in some of the problems and would be modified in' 
the future. 

120 pages, 25 references 



20 



ERIC 



4 



Page 17 



Mills, R.G. (1971) "Probability Processing and Diagnostic 
Search: 20 Alternatives, 500 Trials." Psyc honomic Science. 
24(6):289-292. — 

Ss' capabilities for processing probability information 
in performing a diagnostic search task similar to 
troubleshooting is explored. Six Ss searched (queried) 
among 20 components in a series circuit to isolate a 
single component failure per trial. Failures were 
generated according to 10 alternative 

probability-of-component-failure distributions (D), each 
N = 500. Prior to performing their search on a trial, Ss 
predicted the failure for that trial. Of primary 
interest was the influence of varying D on predictions 
and search performance. Results indicate that: (1) 
distributions of predictions tend -to match actual D in ' 
shape and location but exhibit certain characteristics; 
(2) probability of correct prediction is low and is 
influenced by D; (3) search strategies are suboptimal 
(The optimal solution required an average of 14? fewer 
tests.) but "logical" and to some extent predictable; 
and (U) in some instances, search strategy appears to be 
related to predictions. 

4 pages, 13 references 



Osborn-, W.C. (1970) "An Approach to the Development of Synthetic 
Performance Tests For Use in Training Evaluation." presented at 
the 12th Annual Military Testing Association Conference, French 
Lick, Indiana. 

This paper describes an approach to the selection of 
inexpensive alternatives to fully job-relevant 
performance tests. Based on the concept of content 
validity, the approach is to utilize the skill and 
knowledge required of the trainee as a criterion for 
assessing the cost and job-relevance of fabricated 
alternatives to performance tests. Two job tasks are 
analyzed to illustrate the concept. 

6 pages 
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Rasmussen , J.; Jensen, A. (1974) "Mental Procedures in 
Real-Life Tasks: A Case Study of Electronic Troubleshooting." 
Ergonomics , 17(3) : 293-307. 

An experiment was conducted at a nuclear research 
institute to investigate troubleshooting approaches used 
by skilled electronics repairmen during on-the-job tasks. 
A verbal record of the repairmen's strategies was 
collected for data analysis. Codes were applied to 
describe the heuristics employed: 1) topographic search 
- use the topographic location of measuring points; 2) 
functional search - use the normal functional relation 
between a feature in the system ' s response and a specific 
part in the system; and 3) search by evaluation of a 
fault. This last heuristic implies the human is using a 
mental model of the system relating changes to internal 
signals, parts or components to the changes observed in 
the system response. Results suggest that procedures 
considered unsystematic in a laboratory study may turn 
out to be rational and systematic when one listens to the 
individual's rationalization. Also, it was found .that 
verbal records appear to be useful for recording rapid 
sequences of simple decisions based upon informationally 
redundant observations, 

15 pages, 11 references 
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Rasmussen, J. (1978) Notes on Diagnostic Strategies in Proces s 
Plant Environment . Roskilde, Denmark: Riso National Laboratory. 
Riso-M-1 983. 

In the report are discussed some aspects of state 
Identification and diagnosis in process plant control 
which must be considered in connection with automatic 
disturbance analysis and man-machine interface systems. 

The content of the diagnostic process depends upon the 
overall goal of the diagnostTcian - whether it is to 
protect. the plant, to maintain operation or to repair, 
and it may not necessarily include a determination of the 
cause itself. Important aspects of diagnosis- include 
critical variables and causal flow paths which are 
intimately related to the paths along which events and 
changes propagate through the system, i.e. to the flow 
of energy, matter and information which together form 
complex, interacting flow structures. 

In the process plant environment, a diagnostic task 
implies a search to identify a change from a normal or 
planned plant state. Several elementary strategies can 
be^ identified. In the report a distinction is drawn 
between two main groups - topographic search strategies, 
performed as search through the system with reference to 
a model of normal plant state; and symptomatic search 

s_t.r.at,egi.e.s, , — p e r-f or-m ed — as — a-se a r c h~t h r ou gh— a~l-i-b r'a r y~o-f 

abnormal state models with reference to the actual plant 
state . 



Typical properties of the different strategies are 
discussed such as processing capacity requirements and 
dependence upon a priori analysis. 

The role of the elementary strategies in the overall 
diagnostic task is not discussed since this is considered 
. possible only with reference to selected scenarios 
describing realistic and cofnplex real life situations 
arising from disturbed plant operations. The aim of the 
report is to identify and formulate some of the issues 
which should be taken into account when creating such 
scenarios based on careful field studies and analysis of 
incident reports. Based on these scenarios, different 
overall diagnostic strategies can then be formulated and 
tested by simulator experiments. 

36 pages, 12 references 
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Rigney, J,W, (1973) "A Discussion of Behavioral Technology 

Laboratories CAI Projects in Relation to a CAI Test-Bed Concept . " 

Behavioral Technology Laboratories, Report No. 71, University of 
Southern California, Los Angeles, CA, 

The elements of an instructional system are discussed and. 
some literature bearing on the functions of these 
elements is reviewed, with the objectives of stimulating 
thought about an instructional system as a CAI test-bed 
and of pointing out some results of laboratory research, 
particularly in cognitive psychology, that are believed 
to be worthy of the instructional technologist 's 
attention . * 

In the second part of the report , selected parts of CAI 
projects underway at the Behavioral Technology 
Laboratories are described in relation, to instructional 
system elements, to give an overview of this research. 
The projects include: work on computer grahics to (1) 
simulate front-panel topography and functional 
organization of circuits, (2) facilitate interpretation 
of relative motion, and (3) create visual ana,J,ogies for 
abstract concepts and processes to serve as the: basis for 
students to develop their own internal representations ; 
work on new methods for recording cortical evoked 
potentials and correlating them with learning and memory 

processes; work on a dynamic progra mming model for 

~ aid'ap;uiire ^orTtro'r of pFo & 1 e m - s o IT ilig~Ty pes of C'kl\ ^ and 
work on three-dimensional mock-ups of electronic 
equipment to be placed on-line with a CAI system. 

40 pages, 46 references 




;Rigney, J.W.; Towne, D.M. (1969 ) "Computer Techniques for 
, Analyzing the Microstruc ture of Serial-Action Work in Industry." 
Human Factors , 11 (2?): 113-122. 

Three computer-based techniques for analyzing and 
simulating serial-action tasks are described • The first, 
called BETS (a Bayesian model), measured the efficiency, 
in terms of -expected information, of tests made by 
technicians who -were • troubleshooting malfunctioning 
electronic-circuits. It computed efficiency ratios for a 
technician* s test sequences in comparison to optimal 
sequences. The second method, ARMAN, generated detailed 
time-and-motion analyses from gross descriptions of 
serial-action tasks and man-machine interfaces and 
computed the time costs of these tasks. 

"^•^^^ third technique incorporates a general model of the 
action-goal structure of serial-action work. It operates 
on this structure with special functions composed in a 
symbol manipulation language . This program, called 
TASKSYM, can generate* all alternative correct ways to 
accomplish serial-action work and can track a subject 
through the performance of this work. The model includes 
an .anti-goal structure which identifies act ion sequences 
leading to catastrophic error , e.g., electrical damage to 
the equipment or shock hazard to the performer. TASKSYM 
is extended with LISP to develop TASKTEACH which provides 

f-e.e.dba c k — on — t est s~-a n d-e r-r o rs— a:n d--o f-f e r S"-v a r ro u s— l-eveTs — — 

of help. 

10 pages, 14 references 
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J.W.; Towne, D.M. (1970) "TASKTEACH: A Method for 
^Computer-Assisted Performance Training," Human Factors. 
12(3):285-296. 

A method for using a computer time-sharing system to 
assist the learninp; of serial tasks, from operating 
equipment through troubleshooting, is described. The 
method is based on mediational theory. The current 
resurgence of interest in mediational theory in 
psychological research is noted, and parallels between 
experimental strategies and instructional strategies for 
evoking and manipulating mediating ocesses are pointed 
out. Categories of processes which mediate performance 
of serial tasks are described. Procedures for 
facilitating the learning of these processes are 
implemented by a computer program, called TASKTEACH. The 
program sustains the student^s performance of complex 
serial tasks by giving him variable amounts of support 
while helping him learn and organize the processes which 
mediate his performance of these tasks. The program 
generates output to the student during the learning 
session by processing short lists and the prior responses 
of the student. The lists, which are input to the 
program replace the conventional frame-by-frame 
description of an instructional sequence written in a 
"CAI language 

— 12— pages-,— 2-1— references 
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Rigney, J.W,; Towne, D.M.; King, C,A,; Moran, P,J, (1978) 
"Field Evaluation of the Generalized Maintenance 

TrainiiT-Simulator : I, Fleet Communications System," Behavioral 
Technology Laboratories, Report No. 89, University of Southern 
California, Los Angeles, CA, 

The Generalized Maintenance Trainer-Simulator (GMTS) is a 
concept for giving students in Class C schools intensive 
practice in troubleshooting equipment and systems taught 
in those schools. It can be used for any device in which 
signal paths and their relationships to controls, 
indicators, and test points can be defined. The GMTS 
uses generative CAI . That is , tit generates tne 
interaction with the student by referring to his last 
inputs and to its stored history of interactions with him 
up to that point. To the extent that these individual 
student histories constitute models of individual 
students, GMTS constructs a model of each student to 
interact< with that student. In addition to these 
features, GMTS is uniquely suitable for use in Class C 
school training . The computer program that implements 
the instructional system is indif f erent • to the specific 
equipment being taught. What specific equipment that is 
simulated by thfe GMTS is determined by loading two data 
, bases for that equipment: one describing essential 
internal features of the equipment and the other 
describing the external features. 

This is a report of a field evaluation of the GMTS 
applied to systems level troubleshooting in the UHF 
communications side of the Fleet Communications System. 
Twenty Class A school students waiting to enter C schools 
practiced solving thirty-five troubleshooting * problems. 
Results were generally positive. With respect to 
practice, the students became uniformly more fluent at 
troubleshooting; . mean times to solve a problem were 
decreased by a factor of two and standard deviations of 
these time were decreased by a factor of five. Similar 
reductions existed* for average number of extra 
replacements per problem. Also, students' attitudes 
toward the trainer were generally -favorable. An 
additional field trial is underway, using the AN/SPA-66 
radar repeater as the subject matter. 

85 pages , 3 references 
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Robinson, J.E.,Jr,; Deutsch, W,E,; Rogers, J.G, (1970) "The 
Field Maintenance Interface Between Human Engineering and 
Maintainability Engineering • " Human Factors , 12(3) : 253-259. 

Field maintenance actions reported in a deployed air 
defense system were reviewed to determine and quantify 
(1) the nature of operator and maintainer influences on 
system malfunctions, and (2) the nature of man-machine 
problems being encountered by system personnel in actual 
diagnosis and repair activities. Maintenance events 
described in 213 problem reports from the field were 
categorized according to three probable sources of 
malfunction: primarily human, combination human and 
equipment, and primarily equipment. Human involvement in 
system malfunctions was clearly established in 25% of the 
events reported. Additional information contained in 21 
documents analyzing system maintenance was examined. The 
nature of human influence on system malfunction is 
described by examples of "unfortunate" field maintenance, 
"ordinary" field maintenance, and troubleshooting 
deficiencies. Troubleshooting deficiencies were cited as 
a frequent problem observed by equipment contractors who 
had more knowledge and understanding than the technical 
personnel. The nature of reported man-machine problems^ 
highlighted the fact that human engineering participation' 
in establishing maintainability design requirements can 
serve to make equipment that will be suitable to the 
presence and actions of human users, 

7 pages, 5 references 
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Rouse, W.B., (1973) "Human Problem Solving Perfomance in a Fault 
Diagnosis Task." IEEE Transactions on Systems, Man , and 
Cybernetics , SMC-8 (^7^^8-271": 

It is proposed that humans in automated systems will be 
asked to assume the role of troubleshooter or problem 
solver and that problems which they will be asked to 
solve in such systems wi 11 not be amenable to rote 
solution. The design of visual displays for problem 
solving in such situations is considered, and the results 
of two experimental investigations (N=20) of human 
problem solving performance in the diagnosis of faults in 
graphically displayed network problems are discussed. 
The effects of problem size (9, 25, and H9 components), 
forced-pacing, computer aiding, and training are 
considered. Results indicate that human performance 
deviates from optimality as problem ' size increases. 
Forced-pacing appears to cause the human to adopt fairly 
brute force strategies, as compared to those adopted in 
self-paced situations. Computer aiding substantially 
lessens the number of mistaken diagnoses by performing 
the bookkeeping portions of the task* Further, for 
self-paced situations, skills developed with 
computer-aided training are transferred to unaided 
situations, 

1U pages, 52 references 



Rouse, W,B., (1978) "A Model of Human Decisionmaking in a Fault 
Diagnosis Task." IEEE Transactions on Systems-, Man , and 
Cybernetics . SMC-8 (5Tri5'7-3FT: 

Utilizing elementary concepts from the theory of fuzzy 
sets as well as several nonfuzzy heuristics, a model is 
presented of human decisionmaking in the task of 
troubleshooting graphically displayed networks. The 
performance of the model is compared to the results of 
two pr ev iously reported ex perimental studies , The 
ability of the model to represent human decision making 
as a function of network size, forced-pacing, and 
computer aiding is considered. The model is used to 
support -the hypothesis that humans tend to discount the 
value of information about what has not failed in 
searching for the sources of malfunctions. 

5 pages, 3 references 
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Rouse, W.B. (1979) "A Model of Human Decision Making in Fault 
Diagnosis Tasks that Include Feedback and Redundancy." IEEE 
Transactions on Systems, Man , and Cybernetics . SMG-9 (4) : 237-241 . 

A previously, reported model of human decision making in 
fault diagnosis tasks is extended to include situations 
where feedback and component redundancy are important 
considerations. The model is based on concepts from the 
theory of fuzzy sets. The results of an experiment with 
human subjects are reported and used to estimate the 
parameters of the model. The model is used to illustrate 
the notion that subjects who discount the importance of 
feedback loops tend to have considerable difficulty when 
they encounter problems having a^ large number of loops. 

5 pages, 7 references 



Rouse, W.B. (1979) "Problem Solving Performance of Maintenance 
Trainees in a Fault Diagnosis. Task." Human Factors. 
21 (2): 195-203. 

Forty trainees in an FAA certificate program participated 
in an experimental study of troubleshooting of 
graphically displayed networks. The effects of network 
size, computer aiding, and training were considered. It 
was found that performance degraded as network size 
increased, improved with the use of computer aiding, and 
that skills developed with computer aiding were 
transferred to the unaided situation. 

9 pages, 15 references 



Rouse, W.B. (1979) "Problem Solving Performance of First 
Semester Maintenance Trainees in Two Fault Diagnosis Tasks." 
Human Factors . 21(5). 

Forty-eight first semester trainees in an FAA certificate 
program participated in an experimental study of 
troubleshooting of two different types of graphically 
displayed networks. The effects of network size, redundancy, 
feedback, computer aiding, and training were considered . It 
was found that performance degraded as network size 
increased, degraded as the level of feedback was reduced, 
improved with the use of computer aiding, and that skills 
developed with computer aiding in one task were transferred 
to the other task. 

; 4 references 
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Rouse, W,B.; Rouse, S-H, (1979) "Measures of* Complexity of 
Fault Diagnosis Tasks." to appear in IEEE Transactions on System, 
Man , and Cybernetics , SMC-9 ( 11 ) • 

The literature of complexity is reviewed and the 
distinction between perceptual complexity and problem 
solving complexity is discussed. Within the context of 
two particular fault diagnosis tasks, four measures of 
complexity are considered • These measures are evaluated 
using data from two previously reported experiments which 
employed eighty-eight subjects. It is shown that two 
parti cul ar measures of complexity , one based on 
Information theory and the other based on the number of 
relfjvant relationships within the problem, are reasonably 
good predictors of human performance in fault diagnosis 
tasks. The success of these- measures is explained by the 
fact that they incorporate the human's understanding of 
the • problem and specific solution strategy as well as 
properties of the problem itself. 

54 references 



Saltz, E. ; Moore, J.V.- (1953) A Preliminary Investigation of 
Troubleshooting , Technical Report 53-2. Lackland Air Force 
Base, Texas: Human Resources^ Research Center. 

Troubleshooting was investigated using radar, 
reciprocating engines, and remote control turrets as the 
hardwa»^e for malfunction analysis observation. The 
discrete- types of equipment were chosen in art effort to 
promote generalizability . The following hypotheses were 
formulated and tested: 

1. Good troubleshooters knOw more about the 

functioning of the equipment upon which they 

work than do p'oor troubleshooters. 
2 Good and poor troubleshooters differ in 

previous experience . 
3 . Good and. poor troubleshooters differ in 

intelligence . 

10 pages,. 4- references 
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Saupe, J,L. (195U) Troubleshooting Electronic Equipment : An 
Empirical Approach to the Identification of Certain Requirements 
of £ Maintenance Occupation , Urbana, IL: University oT 
TTlinois . 

The study attempts to identify the criteria for 
proficiency in a troubleshooting situation. The study 
provides "leads for the construction of proficiency tests 
which are important aspects of the efficient selection, 
training, and distribution of maintenance personnel . " The 
goals of the research were: 



1. to develop a psychological, model by which 
to view troubleshooting behavior. 

2. to test certain hypotheses taken from this model 
relative to aspects of the troubleshooting 
process 

3. to illustrate an empirical approach to the 
identification of important objectives 

of training programs 

Subject performance was evaluated on a simulated radio 
receiver and on a paper and pencil test. Empirical'*' 
results were presented.. 



Standlee, L.S.; Bilinskri, C. (1966) "Note on Simulated Vs-. 
Actual Electronic Troubleshooting Performance." Perceptual and 
Motor Skills . 23:532. 

This is a brief note on electronic troubleshooting 
performance^ It discusses the two basic approaches to 
measuring this performance (1) measurement of performance 
on simulated and (2) actual electronic equipment. 114 
subjects (Naval electronics technicians) were divided 
into two groups, one group troubleshooting with the 
simulator, and the other with actual equipment. Results 
indicated poor troublshooting performance with the 
simulator and proficient performance with the actual 
equipment . 

1 page 
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Steinemann, J.H., (I966) "Comparison of Performance on Analogous 
Simulated and Actual Troubleshooting Tasks." U.S. Naval 
Personnel Research Activity, Memorandum SRM 67-1, July I966. 

This investigation compared the performance of a group of 
subjects assessed on a simulated troubleshooting task and 
on the identical actual troubleshooting task using real 
equipment. Subjects were 14 students in the experimental 
training program for Electronics Technicians, conducted 
by the Navy Training Research Laboratory, San Diego. 

Analysis of results revealed that the simulated 
performance measure did not provide a valid estimate of 
performance proficiency on the actual task. Obtained 
negative inter-test correlations indicate that simulated 
test results would actually be misleading in terms of 
estimating actual performance scores. In addition to 
performance score discrepancies*, there were observable 
differences in specific performance procedures and 
overall troubleshooting strategy attributable to the 
differences in test mode* Although the research design 
was weak, the evidence strongly suggests caution in 
assuming that a simulated performance measure, even with 
considerable face validity, will provide a valid estimate 
.of actual performance on a common task. 

12 pages 



33 



Page 30 



Stolurow, L.M.; Bergum, B. ; Hodgson, T. ; Silva, J. (1955) 
"The Efficient Course of Action in » Troubleshooting » as a Joint 
Function of Probability and Cost." Educational and Psychological 
Measurement, 15 (4 ): 462-477. 

This research focuses on the development of a 
probabilistic model of troubleshooting behavior which 
prescribes an efficient course of action in locating 
defects based on probabilities of failures and work time 
given particular symptoms. Data were obtained from 
maintenance records for 46 four-engine aircraft to obtain 
the probability of each possible cause of a particular 
set of diagnostic signs. Mean repair times were also 
completed for each type of repair. A complete method and 
a simplified method are described for determining an 
efficient course of action to be followed in reducing a 
set of possible causes of malfunction to the actual 
cause. The simplified method was proposed as an 
alternative to overcome the cumbersome calculations of- 
the complete method. The simplified method uses the 
ratio of mean repair time to probability of failure to 
determine the course of action to follow in reducing the 
list of possible troubles. Results of the analyses of 
maintenance records were- sufficiently encouraging to 
suggest that furth3r research of this type be done for a 
variety of complex devices, 

16 pages, 4 references 
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Towne, D.M. (1979) "The Automated Integration of Training and 
Aiding Information for t'he Operator/Technician," presented to the 
Third Biennial Maintenance Training and Aiding Conference • 
Orlando, Fla. 

A responsive and intelligent device which can provide 
both training information and job performance aiding can 
be extremely efficient in utilization of hardware, 
software , and so-called course-ware • The hardware 
components useful for presenting automated instruction 
are essentially the same as- those required to aid an 
operator or technician , viz. , a versitile and fast 
Microprocessor CPU, ample random access memory, ample 
bulk storage, and appropriate input-output media. While 
the software driving such a device may be a collection of 
relatively special-purpose routines designed to satisfy 
particular training or aiding needs, a large portion of 
this software- may be common utility functions which 
respond to specific requests of the user or serve to 
manipulate data about the target equipment or data about 
the operator/technician. Finally, the data base which 
characterizes a target equipment can be designed in such 
a way that both training and aiding requirements are met 
without duplication or inefficiency of oreparation 
effort. 

Under a contract with Naval Training Equipment Center, 
Behavioral Technology Laboratories has assembled the 
hardware elements of such a system, designed the target 
equipment, and programmed routines for training novice 
operator/technicians and aiding equipment operation and 
maintenance. 

21 pages, 5 references 
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Wescourt, K.T. ; Hemphill, L. (1978) "Representing and Teaching 
Knowledge for Troubleshooting/Debugging." Institute for 
Mathematical Studies in the Social Sciences, Report No. 292 
Stanford University, Stanford, CA. 

An extensive review of the troubleshooting/debugging 
literature is covered from an artificial intelligence 
perspective. The authors were interested in measuring 
the effect of training, that is, exposure to specific 
debugging heuristics, on problem solving performance. 
Twelve subjects with relatively little programming 
experience participated. They were currently enrolled in 
a BASIC programming course. Subjects were randomly 
assigned to two groups (aided and unaided) based on a 
computer programmer aptitude battery. Aided subjects 
could refer to a text tutorial during problem solving 
sessions. The chronology of subjects' actions were 
judged according to 5 criteria relating to implementation 
of heuristics contained in the tutorial guidelines and 
success of attempted solution. Successful debugging 
grammars that would capture the subjects' strategies were 
not developed. Subjective judgments of participants on 
the research team were necessary to identify the extent 
to which each measure applied to subjects' strategies. 
Results indicated poor performance of both groups in 
solving problems. After the experiment large differences 
were found in post test scores attained on the computer 
programmer aptitude, battery, and thus by chance, subjects 
assigned to the non-aided group had become better 
programmers since completing their course in BASIC. In 
conclusion, there were no strong results to show that the 
tutorial aid affected implementation of heuristics. It 
is suggested that a pedagogy is needed for teaching 
procedural rather than declarative knowledge. 

141 pages, 31 references 
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SECONDARY RESEARCH 



Czeh, R.S. (1957) "Studies of Complex Behavior and their 
Relation' to Troubleshooting in Electronic Equipment." Alexandria, 
VA: Human Resources Research Organi zation • 

This report is a review oT the literature concerned 
either directly or .indirectly with the troubleshooting . of 
electronic equipment . Specifically, this review presents 
the most pertinent findings in the areas of problem 
solving and concept formation and indicates the manner in 
which these findings have impinged upon, or could impinge 
upon thinking about troubleshooting. The literature on 
troubleshooting' is discussed and the final section 
outlines a research task designed to develop an 
efficient, generally applicable procedure fo r 
troubleshooting electronic , and to develop methods for 
teaching the procedure, 

23 pages, 59 references 



Edwards, J.; Norton, S. ; Taylor, S. Weiss, M. ; Dusseldorp, R. 
(November 1975) "How Effective is CAI? A Review of the 
Research." Educational Leadership , 33: 1U7-153* 

This article summarizes many research studies that 
involved Computer Assisted Instruction. The CAI modes 
discussed are: drill and practice, problem- solving , 
simulation, and tutorial. The areas of CAI research 
discussed are: 

1 . CAI as a summplement to traditional instruction 

2. CAI as a substitute for traditional instruction 

3. Effectiveness of different modes of CAI 

^. Comparison of CMI with other nontraditional methods 

of instruction 
5. Compression oT time 
5 . Retention of learning with CAI 
7. Effectiveness according to ability levels 

This literature review referred to 9 studies that showed 
that the combination of CAI and traditional instruction 
was distinctly better than traditional instruction only. 
As a substitute for traditional instruction, the findings 
were mixed with 9 studies finding CAI superior and 8 
studies finding .traditional instruction better. In all 
cases CAI was effective in reducing training time. 



7 pages ,33 references 
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Fattu, N.A. (1956) A Catalog of Troubleshooting Tests . Indiana 
University: Institute of Educational Research, 

This catalog contains examples of various kinds of 
troubleshooting tests. The justification for a catalog 
of troubleshooting tests is that it summarizes and 
illustrates materials assembled from widely scattered 
sources and makes them available to individuals who need 
the information but do not have time to assemble it. 
This catalog attempts to be representative rather than 
exhaustive . 

Tests are reported in the following groups: 

1 . on-the job measures 

2. performance tests 

3* simulator tests 

4. ARC or "tab" tests 

5. pencil-paper and oral-interview tests. 

The major -difference between the troubleshooting research 
of the fifties and the troubleshooting research of the 
seventies is tha use of computers for data gathering in 
the later efforts. 



130 pages, 43 references 
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Fink, CD.; Shriver, E.L. (July 1978) "Simulators for 
Maintenance Training," Air Force Human Resources Laboratory, 
AFHRL-TR-78-27. 

This report reviews past and present applications of 
simulation to maintenance training • Emphasis is on 
describing issues, problems and areas for future research 
as identified by recent authors. There is little 
evidence,^ from^ the literature regarding the degree to 
which simulators promote transfer of training to the 
field.. Thus far, transfer research has demonstrated that 
simulators can increase proficiency during and at the end 
of resident, training.. A variety of research has also 
demonstrated that major variations in resident training 
cannot be correlated with job proficiency after a year or 
more in the field. It still remains to be_ shown 
, conclusively that transfer of training affects 
performance during the first weeks and months on the job. 
If transfer does- occur, it should have its most important 
and noticeable impact during" these weeks . There is a 
need , therefore, for studies whi ch examine the 
relationship between resident training, especially that 
supported by simulators, and on the job proficiency 
during the first one to three months on the' job . 
Allocating the position of simulators and actual 
equipment systems in- maintenance training requires 
research investigating- the optimal sequence of tasks and 
training aids for maintenance training. 

60 pages, 71 references 
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Foley, J. P., Jr • (1977) "Overview of Advanced Systems Division 
Criterion Research (Maintenance) • " Air Force Human Resources 
Laboratory, AFHRL-TR -77-77 . 

A prime but seldom considered cause of the current high 
maintenance cost of DOD hardware, and thus the high 
ownership cost of systems, is the current criteria used 
by personnel systems to select, train, assign, and 
promote maintenance personnel. The current criteria 
emphasize the ability to obtain high scores on 
paper-and-pencil theory and job knowledge tests • This 
paper summarizes the many studies which indicate that 
such tests have little demonstrated relationship to 
ability to perform: job tasks • Table 1 summarizes the 
current research on performance measurement and indicates 
for each citation the low correlations obtained in those 
experiments for various maintenance tasks and 
paper-and-pencil theory tests, job knowledge tests , and 
school grades • Based on an experiment conducted in the 
past by AFHRL, it was shown that graphic symbolic tests 
were more promising than symbolic tests ih measuring job 
task performance, A maintenance task taxonomy is 
discussed and a hierarchy of dependencies is developed 
for electronic maintenance tasks. This conceptualization 
is considered necessary before identifying the relevant 
criteria in job task performance tests. 

22 pages, 38 references 



King, W. (1978) ^'New Concepts in Maintenance Training." Aviation 
Engineering and Maintenance , 2 (6): 24-26. 

This article discusses the present status of Navy 
maintenance training in light of recent developments in 
•CAI (Computer Assisted Instruction). The advantages of 
2-D computer simulators and 3-D real equipment simulators 
for troubleshooting is , the main focus of example. The 
author presents monetary comparisons between actual 
equipment trainers (AET) and simulators of both the 
computer and real equipment typeT The best example is 
the cost comparison for the training of radio room 
technicians for the Trident submarine. The AET costs 20 
million dollars compared to "a 2 million dollar computer 
simulator. King believes that the utilization of 
computer simulations for maintenance training has the ' 
/potential to surpass the savings offered by simulators 
for flight training. 

3 pages: 
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Popham, W.J.; Fattu, N.A. ( 1956) A R eview of 
; : Research . Indiana University: Institute of 

; Educational . Research . 

This report is an annotated bibliography of the 
troubleshooting literature of the fifties. It contains a 
brief description of the work of nearly one hundred 
authors. The report is indexed with the following terms: 
general discussion, proficiency assessment techniques , 
training equipment or materials, analysis of job 
activities, evaluation of maintenance personnel, 
experimentation, bibliography. This report is invaluable 
to persons interested in the early efforts of fault 
diagnosis research. 

127' pages, 110 references 
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g;:, MANUALS OF ACTUAL FAULT DIAGNOSIS OR TRAINING IN FAULT DIAGNOSIS 



AVCO Lycoming. (1975) "AVCO Lycoming 
Troubleshooti-ng Guide." Williarasport , PA: 



Reciprocating Engine 
SSP-475. 



A manufacturer published list of problems, causes, and 
solutions that represents the more common and recurring 
problems encountered by mechanics when troubleshooting. 
The list is general and refers to turbocharged and 
normally aspirated engines. This manual is written for 
the professional mechanic and assumes that shop manuals 
and proper tools are available. 



47 pages 



Bean, R, (1974) Petersen' s Basic Automotive Troubleshooting 
Los Angeles, CA: Petersen PublTsFiTng Zo~. "~ 

This manual is a very basic guide for the novice or 
do-it-yourself automotive-, enthusiast. It includes an 
easy to follow symptom-cause- format with illustrations. 
The book is divided into, such chapters as the following: 
Troubleshooting the Brakes;. Troubleshooting the Ignition 
System; Troublshooting Automatic Transmissions; Engine 
Oil Pressure Problems; and Troubleshooting with a Vacuum 
or Compression Gauge. Simple logic trees are included 
for .the starting,, ignition and fuel, systems. 

144 pages 
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CKilton Book Co.. (1972) "Troubleshooting," Chilton' s Mustang 
Repair & Tune-Up Guide. Radnor, .PA: Chilton Book Company: 
31-45. 

A section of the manual is designed to aid in the rapid 
diagnoisis of engine problems. The systematic format is 
used to diagnose problems ranging from engine starting 
difficulties to. the need for engine overhaul. It is 
assumed that the user is equipped with basic hand tools 
and test equipment (tach-dwell meter, timing light, 
voltmeter, and ohmmeter). Troubleshooting is divided 
into two .sections . The first. General Diagnosis , is used 
to locate the problem area. In the second. Specific 
Diagnosis , the problem is systematically evaluated. The 
format presents a malfunction fjystem followed by a list 
of tests and procedures. The proper results of the test 
.are provided in an effort, to aid the troubleshooter . 

15 pages 



Glenn, H^T. (1971) Glenn' s Auto Troubleshoot ing Guide. 
Philadelphia, PA: Chilton Book CoZ ' 

This manual is another example of a popular book written 
for the automotive enthusiast rather than the experienced 
mechanic. The specific areas that .it covers are: 
emergencies, hard starting, engine failure, excessive oil 
consumption, engine noises, overheating, fuel system 
trouble-, electrical system failure, ignition system 
failure, drive, transmission, and rear axle trouble, and 
power brake and power steering failure. 

98 pages 



Howard W. Sams & Co. (1973) Tune-up Service with Testing, 
Diagnosis and Troubleshooting- IndTTahapolis , IN: Howard W. Sams 
& Co . , Inc . 2nd Edition. 

This manual covers the latest emission control systems, 
electronic ignition systems, electronic test equipment, 
and conventional engine analysis equipment. The 
troubleshooting . section presents -the familiar 
symptom/cause format. 

72 pages 
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Mitchell Manuals, Inc. (1977) Diagnostic Guide . San Diego, CA 
Mitchell Manuals, Inc. ' 

The book contains information designed to aid the 
professional mechanic in diagnosis and repair of unusual 
automotive problems. Since many malf unctiojis are caused 
by incorrect , inoperative , defective or missing 
components, attention is focused on these problems and 
conditions. All information presented has been carefully 
compiled , primarily from- factory service bulletins , 
pinpointing the cause of specific malfunctions . This 
diagnostic guide is designed to help the mechanic avoid 
potential problems and save time. 

The book is arranged for quick reference in finding 
particular information. The book's main divisions are 
done by vehicle manufacturers: American Motors, Buick, 
Cadillac, etc. Within each of the divisions there are 
three areas of information: 

1) troubleshooting and diagnosis 

2) jacking, hoisting and towing 

3) hood latch and safety release locations 

1U2 pages 



General Motors Corporation. (1978) Diagnosis and Repair Manual 
Easy to Follow Graphic Troubleshooting Procedures. Southfield 
MI: General Motors Corporation. 

This manual contains graphic mode diagnosis and repair 
charts. They are a relatively new approach to 
systematic diagnosis and repair. The charts are easy to 
use since they require minimal reading. The sketches of 
exact engine parts allows the mechanic to proceed to 
work without reading a manual, although on complex 
repairs the book emphasizes that a manual should be 
used. 

These diagnosis procedures apply to various automotive 
systems in most 1970 thru current GM cars and light-duty 
trucks-, with, some procedures applying to even earlier 
model years. The procedures are based on the fact that 
the vehicle worked right at one time and the problem is 
^^^^^^^^^ d^^^ 

201 pages 
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MATERIALS INDIRECTLY RELATED TO FAULT DIAGNOSIS 



Berner, W.S.; Hamilton, L.A.; Best, W.R. (1974) "A New 
Approach to Evaluating Problem-Solving in Medical Students." 
Journal of Medical Education . 49:666-672. 

This paper describes the development of a new technique 
to assess clinical, problem-solving skills in medical 
students. The procedure enables each step in the 
solution of clinical - problems to be evaluated 
independently by providing feedback to students .at each 
decision point. There are several unique features, 
including the incorporation of some aspects of the 
problem-oriented medical record. Students were required 
to read an extensive data base and construct problem 
lists, order diagnostic tests, and plan for the 
management of the patient. The technique was designed to 
(a) be administered to large groups of students, (b) be 
-easily scorable, ^ and (c) . overcome some of the 
difficulties inherent in previously developed mean^ of 
assessment . 

7 pages, 11 references 
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Brown, J.S.; Burton, R.R. (1978) "Diagnostic Models for 
Procedural Bugs in Basic Mathematical Skills Cognitive Science. 
2(2): — ^ 



A new diagnostic modeling system for automatically 
synthesizing a deep-structure model of a student's 
misconceptions of bugs in his basic mathematical skills 
provides a mechanism for explaining why a student is 
making a mistake as opposed to simply identifying the 
mistake. This report, is divided in four . sect ions : The 
first provides examples of the problems that must be 
handled by a diagnostic model* It then introduces 
.procedural networks as a general framework for 
representing the knowledge underlying a skill. The 
challenge in designing this representation is to find one 
that facilitates the discovery of misconceptions of bugs 
existing in a particular student's encoding of this 
knowledge. The second section discusses some of the 
pedagogical issues that have emerged from the use of 
diagnostic models within an instructional system. This 
discussion is framed in the context of. a computer-based 
tutoring/gaming system developed to teach students and 
student teachers how to diagnose bugs strategically as 
well as how to- provide a better understanding of the 
underlying structure of arithmetic skills. The third 
section describes our uses of an executable network as a 
tool for automatically diagnosing student behavior, for 
automatically generating "diagnostic" tests, and for 
judging the diagnostic quality of a given exam. Icluded 
in this section is a discussion of the success of this 
system in diagnosing 1300 school students from a data 
base of 20,000 test items. The last section discusses 
future, research directions. 

38 pages, 30 references 
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Ellery, G.L. (1977) "The EE's Role in the Garage Diagnosis of 
Vehicle Faults." Proceedings of 1978 IEEE Conference on Vehicular 
Technology, Denver. *f 

• The complexity of modern automobiles, trucks and other 
internal combustion engines demands more sophisticated, 
computerized diagnostic equipment. The requirement of 
high fuel economy and low hydrocarbon emissions further 
highlights this need. The author -discusses the fact that 
automobile manufacturers are increasing their involvement 
in the design and construction of automated test 
equipment. The R & D for such equipment is the combined 
effort of the EE, ME, and Automotive Service Engineer. 

Computer controlled test equipment which relies on 
micro-processor technology is already found in many 
automotive repair facilities, but the sophistication of 
such .equipments is rapidly increasing. The author offers 
examples of electrical analysis of starter-motor-current 
oscillation and analysis of ignition spark amplitude to 
determine, engine compression. 

3 pages • 



McGuire, C.H. ; Babbott, D. (1967) "Simulation Technique in. the 
Measurement of Problem-Solving Skills." Journal of Educational 
Measurement . 4:1-10. 

Employing simulation technique , analogous to that utilized 
in management games and military exercises, the authors 
have directed the development of a series of branched 
problems in patient management requiring sequential 
analysis and decision. These problems are designed tf^ 
measure aspects of behavior 'defined by a criterion gr .up 
as essential components of clinical • competence . The 
materials are- suitable to either paper and pencil or 
on-line application. The validity and reliability, of 
simulations is also discussed.. 

10 pages, 13 references 
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■McGuire, C^H,; Solomon, L.M,; Bashook, P,G. (19T6). 
Construction ai^ Use of Written Simulation , United States of 
America: The Psychological Corporation • 

This work is a comprehensive discussion of written 
simulations • The discussion ranges from basic 

definitions through construction, use and evaluation of 
simulations. The book is additionally applicable to 
computer simulations, or real equipment simulations • 
Sample problems are presented throughout the book. 



Murray, T.S.; Cupples, R.W.; Barber, J.H,; Dunn, W.R.;. Scott, 
D.B.; Hannay, D.R. (1977) "Teaching Decision Making to Medical 
Undergraduates by Computer-Assi sted Learning," Medical Education. 
11:262-264. 

This paper describes a computer-assisted learning (CAL) 
program that is used, on an open-acess basis by students" 
at the University of Glasgow medical school. The authors 
report an experiment in which the ex perimental group 
(n=12) was presented with on-line case studies in which 
they had to decide- on patient treatment. Upon completion 
of the experiment, the experimental group scored higher 
on a factual knowledge test and had improved decision 
making skills based on comparison of their scores to 
those of medical experts. The subjects felt that CAL 
enabled them to consider and discuss clinical decisions 
in a way that is both more realistic and more instructive 
than is possible in the more orthodox clinical teaching. 

3 pages, 9 references 
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Swain, A.D. ( 1 96 1) Factors Affecting Degree of Automation in 
Test and Checkout Equipment Stamford; Conn.: Dunlap ana" 
Associates , Inc . 

.This report deals primarily with the human factors 
problems attendant to the increasing automation of weapon 
system; test and checkout actions. The purpose of the 
report is to identify unfavorable taitClaal situations 
resulting from less-than-optimjjra allocation of test and. 
checkout actions, 'from ina^Bequate design of automatic 
checkout equipment ( ACE.) , or from particular management 
or other practices related to the utilization of ACE- 
These cases suggest ways of avoiding such situations in 
the future. While most of the examples are taken from 
Air Force weapon systems, some are taken from Army, and 
Navy weapon systems and equipment; and the principles 
and guidelines are generally applicable ♦ to all the 
services. It is only the application which should be 
different, as considerable differences often occur 
between services in such areas as training, manpower 
selection, environmental conditions, and logistics 
support (to name only a few -differences in system 
criteria) . 

22U pages 
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Wollmer , R.D. ; Bond, .N^A., Jr. (1975) "Evaluation of a 
Markov-Decision Model for Instructional Sequence Opt imization. " 
Behavioral Technology Laboratories, Report No. 76 , University of 
Southern California , Los Angeles, CA. 

Two CAI programs in electronics and trigonometry were 
written to .test the Wollmer Markov Model for optimizing 
hierarchical learning; calibration samples totalling 110 
students ■ completed these programs. Since the model 
postulated that transfer effects would be a function of 
amount of practice, half the students were required to* 
complete one practice problem successfully before moving 
to the next stage; the other half* had to do two practice 
problems successfully. 

All students completed the courses satisfactorily. 
Practice effects were small ; students who had one 
success in each stage did about as well as those who - had 
two successes. The Wollmer model was thus not suitable 
for optimizing instruction, in terms of minimizing 
overall time, in these partioul-^'^ courses. Perhaps the 
main reason for this i , 3ult w.. s that, as the student 
works up to the top of the hierarchy , the sheer number of 
subskills involved in the final task becomes a major 
determinant of performance time, and number of practice 
trials has a relatively minor effect, unless a very large 
number of practice trials is given. To see if a much 
larger amount of practice would affect time to perform, 
three additional subjects were- run with the electronics 
problems under the condition that 5 correct solutions 
were required at each of 11 levels. It was found that a 
much larger training time was required to get a 2-3 
minute improvement at the final level. Because of this 
relative^ inefficiency, the Wollmer model would not 
prescribe such extensive additional, practice. 

35 pages, 8 references 
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